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Instructions: The information contained in this document is COMPANY CONFIDENTIAL arib may not be disclosed to others without prior 
authorization. Submit this disclosure to the HP Legal Department as soon as possible. No patent protection is possible until a patent application is 
authorized, prepared, and submitted to the Government. — 



Descriptive Title of Invention: 

Pointing device tor digital camera displays 



Name of Project: Perceptual Image Quality 



Product Name or Number 



Was a description of the invention published, or are you planning to publish? If so, the date(s) and publication(s): 
No 

Was a product including the invention announced, offered for sale, sold, or is such activity proposed? If so, the date(s) and locations): 
No 

Was the invention disclosed to anyone outside of HP t or will such disclosure occur? If so, the date(s) and name(s): 
No 

if any of the above situations will occur wrimin 3 months, caff your IP attorney or the Legal Department now at 1-657-2542 or 415*857-2542 



Was the invention described in a lab book or other record? if so, please identify (lab book #, etc.) 
2214-22 



Was the invention built or tested? If so, the date: 
9/20/98 



Was this invention made under a government contract? If so, the agency and contract number: 
No 



Description of Invention: Please preserve all records of the invention and attach additional pages for the following. Each additional page should 

be signed and dated by the 'mventor(s) and witness(es). 
A Prior solutions and their disadvantages (if available, attach copies of product literature, technical articles, patents, etc.). 

B. Problems solved by the invention. 

C. Advantages of the invention over what has been done before. 

D. Description of the construction and operation of the invention (include appropriate schematic, block, & timing diagrams; drawings; samples; 



grapns; Tiowcnans; computer iisungs, test results, eiu.j 
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Description of Invention: Please preserve Records of the invention and attach additional pages to* the following. Each additional page should 
be signed and dated by the inventor(s) and whness(es). 

A. Prior solutions and their disadvantages (if available, attach copies of product literature, technical articles, patents, etc). 

B. Problems solved by the invention. 

C. Advantages of the invention over what has been done before. 

D. Description of the construction and operation of the invention (include appropriate schematic, block. & timing diagrams; drawings; samples; 
graphs; flowcharts; computer listings; test results; etc.) 

The invention described here allows a digital camera to be used as a pointing device. The motion of the 
camera is detected, and the motion of the camera is used to position graphic elements on the camera's own 
display. The camera's motion can be detected with sensors, such as gyroscopes, or the camera itself can be used 
as a motion sensor. One application of this involves using the camera as a computer mouse, or like a gun-sight, 
to select images from a sheet of low-resolution ("thumbnail") images. The motion of the camera is tracked, and 
the user aims at the desired image from a sheet of thumbnails. This application is illustrated in figures 1 and 2. 1 
have implemented this using optical motion tracking software developed by Andrew Patti. 



A. Prior solutions: Digital cameras have not generally used pointing devices to interact with the interface 
presented on the display. Typically, they use a set of push buttons to step through menus and to select 
images from memory for display and deletion. New digital cameras may rely on micro display devices, 
instead of the now more common panel display. To view a micro display, the user needs to hold the display 
close to his or her eye and view the display through an eyepiece. This arrangement makes the interface 
controls more difficult to use, since the user will not be able to see the controls while viewing the display. 

B. Problems solved: The present invention allows the user to interact with the information display in a way 
similar to the computer mouse. In the present invention, the entire body of the camera is moved, and the 
movement of the camera is recorded. The motion information can then be used to position a Cursor, or to 
position graphic elements. For example, graphic elements can be positioned so they stay fixed relative to the 
world as the user moves the camera. 

C. Advantages of the new method: With most digital still cameras, the user can load a previously captured 
image to the display by selecting it from a grid of low-resolution ("thumbnail") images. The thumbnail is 
selected by pressing buttons that move a cursor across the thumbnails until the desired picture is under the 
cursor. With the new method, the user can look into a micro display and will be presented with the 
thumbnails. A computer can continuously reposition the thumbnails so they appear to be fixed relative to the 
world. The user can then select a thumbnail by simply pointing the camera at the desired thumbnail. 

D. Description of the invention: In the present implementation, the position of the camera is tracked by 
optical flow. The camera records a sequence of images. By comparing the images with each other, the 
motion of the camera can be estimated. Detgirnining the motion of the camera by comparing sequential 
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images taken with the camera i; , %vell described in the literature, and thiv«^proach has the advantage of not 
requiring any additional hardware. Alternative implementations could use sensors such as gyroscopes, tilt 
sensors, compasses, etc to measure the position of the camera. These solutions would be more robust, but 
may be more expensive to implement. 





Figure 1. Using the camera like a mouse. 



A sheet of thumbnail images is shown superimposed on a view through the camera (upper 
figure). The cross is a cursor, and this cursor is always fixed relative to the camera. With 
motion tracking, the sheet of thumbnail images is fixed relative to the world. That is, the 
computer constantly repositions the sheet as the camera moves, so the sheet seems to be fixed 
in position relative to the objects seen through the camera. When the user moves the camera 
(lower figure), the cursor moves relative to the world and to the world-fixed thumbnails. This 
allows the user to select an image by simply pointing the camera at the desired image. The 
camera itself is used as the only pointing device. No other mouse, joystick or other device was 
used to move the cursor in this figm^JTie software use£ is in the attached listing. 
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view . 



Figure 2. How it works 



Motion tracking software (developed by Andrew Patti eL al.), tracks the motion of image 
features in the scene. For example, the image of the computer monitor on the desk moved 
down and slightly right between frame 1 and frame 2. The sheet of thumbnails is moved by the 
same amount and in the same direction as the features in the scene. The sheet thus appears to 
be motionless relative to the objects in the scene (such as the computer monitor). 

Since the thumbnails appear fixed relative to the world, the user can use the camera like a 
gun-sight. The crosshairs stay fixed relative lo the camera, and the sheet of thumbnails stays 
fixed relative to the world, so the user can place the crosshairs on the desired thumbnail by 
simply aiming the camera. 
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